Increased reproducibility of machine learning research has been a driving force for dramatic improvements in learning performances. The scientific community further fosters this effort by including reproducibility ratings in reviewer forms and considering them as a crucial factor for the overall evaluation of papers. Accompanying source code is not sufficient to make a work reproducible. The shared codes should meet the ML reproducibility checklist as well. This work aims to support reproducibility evaluations of papers with source codes. We propose an end-to-end system that operates on the Readme file of the source code repositories. The system checks the compliance of a given Readme to a template proposed by a widely used platform for sharing source codes of research. Our system generates scores based on a custom function to combine section scores. We also train a hierarchical transformer model to assign a class label to a given Readme. The experimental results show that the section similarity-based system performs better than the hierarchical transformer. Moreover, it has an advantage regarding explainability since one can directly relate the score to the sections of Readme files.

Source code is essential for researchers to reproduce the methods and replicate the results of artificial intelligence (AI) papers. Some organizations and researchers manually collect AI papers with available source code to contribute to the AI community. However, manual collection is a labor-intensive and time-consuming task. To address this issue, we propose a method to automatically identify papers with available source code and extract their source code repository URLs. With this method, we find that 20.5% of regular papers of 10 top AI conferences published from 2010 to 2019 are identified as papers with available source code and that 8.1% of these source code repositories are no longer accessible. We also create the XMU NLP Lab README Dataset, the largest dataset of labeled README files for source code document research. Through this dataset, we have discovered that quite a few README files have no installation instructions or usage tutorials provided. Further, a large-scale comprehensive statistical analysis is made for a general picture of the source code of AI conference papers. The proposed solution can also go beyond AI conference papers to analyze other scientific papers from both journals and conferences to shed light on more domains.

"Relying on common solutions in places where there's no point in reinventing the wheel has been one of the mainstays of the Estonian digital state," Government Chief Information Officer (CIO) and Deputy Secretary-General for IT and Telecom Siim Sikkut said in a Ministry of Economic Affairs and Communications press release. "This is how X-Road and the digital identity were born, for instance, which made developing e-services several times faster and easier for everyone," Sikkut continued. "Now we want to bring the same platform-based approach and acceleration into the field of AI, and I'm glad that the first step in that direction has been taken." The first base component for AI-based solutions added to the source code repository is a text analysis tool created by Texta OÜ, which has been used by many institutions to date for increasing the effectiveness of their work processes and the automation of routine activities. The Ministry of Education and Research, for example, uses the tool for the audit of document management aimed at identifying documents which have gone public without permission.